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For  military  systems,  environmental  stressors  (e.g.  motion,  temperature,  noise)  must  be 
considered  during  decision  making  related  to  manpower  requirements,  workload 
determination,  design  tradeoffs,  and  mission  effectiveness/sustainability  early  into  and 
throughout  the  system  acquisition  process.  Current  human  performance  modeling 
techniques  may  have  limited  predictive  utility  and  have  not  been  fully  validated  against 
operational  human  in  the  loop  (HIL)  data.  As  a  result,  they  may  lack  sufficient  fidelity  to 
support  systems  engineering  needs  to  predict  the  individual  and  interactive  effects  that 
environmental  stressors  may  have  on  human  performance.  The  purpose  of  this  paper  is  to 
describe  an  approach  for  developing  performance  shaping  function  (PSF)  algorithms  for 
environmental  stressors  that  can  be  integrated  into  human  performance  modeling  tools. 
These  high  fidelity  plug-in  algorithms  are  anticipated  to  provide  an  enhanced  level  of 
predictive  validity  when  compared  to  current  discrete  event  modeling  tools.  The 
algorithms  will  address  environmentally  induced  limitations  that  are  levied  on  human 
performance  and  enhance  decision  making  in  defense  acquisition  system  design  and  cost 
versus  performance  tradeoffs. 

INTRODUCTION 


Within  the  current  defense  acquisition  design 
climate,  realization  of  reduced  manning  initiatives 
must  be  tempered  by  limitations  of  human 
performance  that  may  affect  mission  effectiveness, 
sustainability,  and  safety.  Environmental  factors 
may  induce  onset  of  a  number  of  physiological  and 
biomechanical  events  that  can  quickly  reduce 
human  capabilities  to  a  fraction  of  what  they  would 
be  within  a  stable  environment.  For  example,  ship 
motions  can  limit  a  crews’  ability  to  perform 
essential  command,  control,  and  communications 
functions,  navigation  tasks,  and  emergency 
procedures  (Stevens  &  Parsons,  2002). 

Currently,  validated  impacts  of  environmental 
stressors  (e.g.,  motion,  temperature,  fatigue)  on 
human  performance  are  not  fully  considered  during 
defense  system  design  tradeoffs,  workload 
estimation,  and  manpower  determination.  Further, 
algorithms  used  for  human  performance  modeling 
may  have  limited  predictive  utility  and  have  not 
been  fully  validated  against  operational  human 


performance  data.  As  a  result,  they  may  lack 
sufficient  fidelity  to  accurately  predict  the 
individual  and  interactive  effects  that  these  stressors 
have  on  human  performance. 

Assessing  the  effect  of  these  stressors  on  human 
performance  is  not  a  novel  idea;  there  are  currently 
many  modeling  tools  designed  specifically  for  this 
purpose.  All  modeling  tools,  however,  are 
dependent  upon  the  quality  and  fidelity  of  the 
algorithms  embedded  within  them.  The  current 
effort  aims  to  enhance  the  predictive  capability  of 
these  algorithms. 

The  user’s  guide  of  one  such  modeling  tool. 
Improved  Performance  Research  Integration  Tool 
(IMPRINT),  states  the  need  for  updating  and  adding 
new  algorithms  as  something  that  should  be 
considered  for  improving  utility  (Alton  Science  & 
Technology,  2009).  This  enhancement  of  human 
performance  shaping  function  (PSF)  algorithms 
would  provide  a  basis  for  improved  decision 
making  early  into  and  throughout  the  defense 
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acquisition  design  process.  The  current  effort  can 
help  enhance  the  total  system  picture  (i.e., 
hardware,  software,  human,  and  environment)  by 
enhancing  accuracy  of  estimates  for  task  time,  task 
accuracy,  manpower,  workload,  fatigue, 
effectiveness,  and  sustainability.  This  integrated 
picture  will  result  in  more  precise  predictions  of 
total  system  performance  and  ownership  cost  and 
could  reduce  the  need  for  costly  redesign  by 
considering  environmentally  induced  limitations  to 
human  performance  during  the  design  process. 

HUMAN  PERFORMANCE  MODELING 

An  Office  of  Naval  Research  (ONR)  funded 
research  effort  consisting  of  team  members  from 
government,  industry,  and  academia  is  currently 
underway  to  develop  processes  for  improving  the 
predictive  capabilities  of  PSF’s.  The  goals  of  this 
effort  are  threefold;  increase  fidelity  of  existing 
PSFs,  create  new  PSFs,  and  implement  PSFs  within 
discrete  event  modeling  tools  to  improve 
predictiveness  of  human  performance  within  the 
presence  of  environmental  stressors.  The  PSFs  will 
be  applied  in  the  form  of  mathematical  algorithms 
and  implemented  as  plug-ins  to  relevant  task 
network  models  within  off-  the-shelf  software. 

The  initial  modeling  effort  is  relevant  to  tasks 
performed  aboard  a  Navy  ship.  The  modeling 
environment,  or  implementation  software,  will  be 
chosen  based  upon  the  type  of  human  performance 
question  being  asked.  Some  examples  of  potential 
candidates  for  the  modeling  environment  are 
described  briefly  below. 

IMPRINT  can  be  used  to  apply  environmental 
stressor  algorithms  in  real  time  during  a  mission 
simulation,  or  allow  the  insertion  of  a 
preprogrammed  stressor  into  a  model  (Alion 
Science  and  Technology,  2009).  The  software  uses 
these  encoded  algorithms  to  determine  the  impact  of 
environmental  stressors  (e.g.,  heat,  cold,  motion)  on 
task  time  and  accuracy.  IMPRINT  is  used  primarily 
to  analyze  crew  effectiveness  and  predict  human 
performance  decrements. 

Alion  has  also  developed  the  Total  Crew  Model 
(TCM)  is  used  to  assess  the  adequacy  of  proposed 
manpower  levels  by  evaluating  system 
effectiveness.  TCM  can  evaluate  task  throughput  by 
assigning  functions  to  individuals  while  applying 


constraints,  such  as  fatigue,  to  manpower.  The 
output  may  include  a  timeline  of  allocated  tasks, 
time  spent  on  tasks,  task  failures,  or  fatigue  levels. 

A  third  modeling  tool  is  one  which  allows  the 
user  to  schedule  appropriate  numbers  of  crew 
members  and  their  sleep/wake  cycles  to  generate 
optimized  watchbills.  The  Fatigue  Avoidance 
Scheduling  Tool  (FAST),  developed  by  the  United 
States  Air  Force,  utilizes  a  biomathematical  fatigue 
model  and  individualized  performance  prediction 
algorithms  to  quantify  the  effects  of  various  work- 
rest  schedules  on  human  performance.  The  graphic 
input-output  display  shows  cognitive  performance 
effectiveness  as  a  function  of  time  (Eddy  &  Hursh 
2001). 

Each  of  these  modeling  tools  can  be  used  to 
accomplish  different  human  performance  modeling 
objectives;  however,  they  all  rely  upon  one  common 
thread.  In  order  for  them  to  predict  human 
performance  impacts  accurately  they  must  utilize 
valid  task  networks  and  performance  shaping 
algorithms  as  input.  The  models’  output  can  then  be 
compared  to  human  in  the  loop  (HIE)  data  to  assess 
validity  against  an  established  benchmark.  As  the 
adage  goes,  “garbage  in,  garbage  out”  hence,  higher 
fidelity  data  and  algorithms  yield  more  accurate 
predictions.  The  Naval  Air  Warfare  Center  Training 
Systems  Division  (NAWCTSD)  Human  Systems 
Integration  (HSI)  PSE  Team  set  out  to  improve  the 
outputs  of  predictive  human  performance  modeling 
by  increasing  the  fidelity  and  validity  of  these 
algorithms. 

APPROACH 

The  algorithm  development  approach  adopted 
by  the  team  involved  obtaining  data  relevant  to 
individual  and  interactive  impacts  of  environmental 
stressors  on  human  performance,  development  of 
human  performance  models,  development  of  PSP 
algorithms,  and  validation  of  modeled  output 
against  HIE  performance  data. 

Obtain  Data 

An  extensive  literature  review  was  conducted  to 
assess  the  current  state  of  research  related  to  the 
impact  of  a  range  of  environmental  stressors  on 
human  performance.  Operational  definitions  were 
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refined  for  environmental  stressors  ineluding 
illumination,  motion,  vibration,  fatigue,  G-foree, 
and  temperature. 

Over  250  environmental  stressor  doeuments 
(e.g.,  experimental  studies,  literature  reviews,  meta¬ 
analyses)  were  reviewed  and  relevant  data  were 
extraeted  and  arranged  into  a  eomprehensive 
database  that  ineluded  independent  and  dependent 
variables,  statistical  results,  effect  sizes,  and 
pertinent  notations  explicit  to  the  validity  of  the 
study.  Topical  content  addressed  topics  such  as 
vehicle  motion,  motion  sickness,  uncoupled  motion, 
whole-body  vibration,  multi-axis  vibration, 
combinations  of  vibration  frequencies  and 
amplitudes,  sleep  deprivation,  shift  work,  thermal 
regulation,  thermal  comfort,  ambient  temperatures, 
illumination,  sustained  acceleration,  workload,  and 
fatigue.  Based  on  ONR  guidance,  data  availability, 
and  preliminary  analyses  of  programmatic 
requirements  the  initial  stressors  of  focus  were 
determined  to  be  motion,  fatigue,  and  thermal 
stress. 

Develop  Models 

Fundamental  to  algorithm  development  is  the 
iterative  process  of  integrating  performance  data 
into  task  network  models.  Task  data  were  derived 
from  documentation  for  our  initial  transition 
platform  (e.g.,  test  reports,  risk  assessments,  hazard 
control  records,  and  training  documents).  Data 
collected  and  analyzed  were  aligned  with  previous 
and  ongoing  research  to  create  initial  task  network 
models.  Gaps  in  the  literature  and  available  data 
were  indentified  and  are  being  supplemented 
through  implementation  of  a  data  acquisition 
approach  that  includes  additional  experimental  and 
operational  evaluations  designed  to  bridge  these 
gaps. 

Based  upon  analysis  of  ship  design 
documentation,  ONR  guidance,  data  availability, 
and  SME  input  it  was  determined  that  motion  would 
be  chosen  as  the  initial  stressor  of  interest.  It  was 
also  decided  that  the  initial  modeling  effort  would 
be  focused  on  the  manual  material  handling  (MMH) 
aspect  of  vehicle  launch  and  recovery  (L&R).  It  is 
anticipated  that  there  will  be  a  more  prevalent  effect 
of  motion  on  manual  tasks  vice  cognitive  tasks. 
Therefore,  this  direction  may  provide  a  more 


sensitive  model  within  which  to  investigate  the 
approach,  processes,  and  algorithm  application. 

Based  on  the  question  being  asked  (e.g.,  how 
many  people  does  this  task  take;  how  effective  is 
this  crew)  and  the  software  capabilities  mentioned 
previously,  the  appropriate  modeling  environment 
can  be  chosen.  Once  the  models  are  constructed, 
algorithms  are  plugged  into  the  modeling 
architecture  (e.g.,  IMPRINT,  TCM)  and  applied  to 
the  task  networks  for  human  performance  effects. 
Impacts  are  evaluated  in  terms  of  variables  such  as 
task  time,  task  accuracy,  manpower,  workload, 
fatigue,  effectiveness,  and  sustainability  of  tasks. 

Develop  Algorithms 

The  algorithm  component  development 
approach  is  based  upon  findings  and  data  obtained 
from  laboratory,  simulation,  and  HIL  experimental 
data.  Critical  algorithm  component  determination 
involves  consideration  of  the  potential  influence  of 
several  confounding  and  affecting  variables  on 
human  performance.  For  example,  it  is  anticipated 
that  the  magnitude  and  direction  of  observed  effects 
may  be  influenced  by  task/stressor  type, 
task/stressor  duration,  or  stressor  dosage  (Cheung, 
Brooks,  &  Hoofer,  2002;  Hancock,  Ross,  &  Szalma, 
2007;  Hancock  &  Vasmatzidis,  2003;  Muth,  2006; 
Pilcher,  Nadler,  &  Busch,  2002;  Stevens  &  Parsons, 
2002).  Hence,  not  all  tasks  are  equally  demanding 
and  some  tasks  may  be  more  susceptible  to  stressors 
than  others.  Additionally,  there  is  inherent 
variability  between  individuals  (e.g.,  gender,  age, 
and  exposure  history)  and  therefore  some  may  be 
more  susceptible  to  environmental  stressor  exposure 
than  others  (Bos,  Damala,  Lewis,  Ganguly,  & 

Turan,  2007;  Stevens  &  Parsons,  2002). 

Efforts  are  being  aimed  toward  identifying  as 
many  influencing  factors  as  possible  and 
determining  gaps  for  representative  data,  constants, 
equations,  performance  curves,  or  ranges.  Some  of 
these  factors  include  platform  location,  adaptation, 
susceptibility  score,  onset  cues,  recovery  times,  etc. 
Initially,  base  algorithms  are  being  developed  which 
will  be  enhanced  iteratively  as  additional  data  is 
acquired  and  analyses  are  performed.  Since  initial 
modeling  is  being  focused  on  motion  stressor 
effects,  an  example  of  the  current  state  of  a  base 
equation  for  the  motion  stressor  is  presented  below 
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HP  (ss)  =/(TT  X  TD  X  D  X  I  X  MSI  x  MSDV  x  PH(z)) 

where  HP  is  human  performanee,  SS  is  sea  state, 

TT  is  task  type,  TD  is  task  diffieulty,  D  is  duration 
of  exposure,  I  is  individual  differenees,  MSI  is 
motion  siekness  ineidenee,  MSDV  is  motion 
siekness  dose  value,  PH  is  plaeeholder,  and  z  is 
human  performanee  faetor.  The  relationships  and 
weightings  of  the  eomponent  effeets  are  still  being 
determined,  these  base  formula  examples  are  not 
meant  to  imply  a  multiplieative  relationship. 

The  MSI  and  MSDV  eomponents  were 
ineorporated  to  prediet  the  oeeurrenee  of  motion 
siekness  based  on  the  amplitude,  frequeney,  and 
duration  of  exposure  to  ship  motions.  There  are  two 
existing  models  for  predieting  the  oeeurrenee  of 
motion  sickness:  the  MSI  Model  and  the  Vomiting 
Incidence  (VI)  Model  (Stevens  &  Parsons,  2002). 
Both  models  express  findings  in  units  of  percent  of 
the  population  that  has  vomited  after  exposure  of  a 
specified  duration.  A  “motion  sickness  dose  value” 
is  also  defined,  which  may  be  used  to  predict  the 
percentage  of  persons  likely  to  vomit  after  exposure 
to  known  magnitudes  and  durations  of  vertical 
oscillation  in  the  frequency  range  0.4  to  0.5  Hz.  The 
motion  sickness  dose  value  is  defined  as: 

MSDV^  =  (t)dt^ 

where  az  is  the  frequency-weighted  z-axis 
acceleration  and  T  is  the  period.  Using  the  motion 
sickness  dose  value,  the  actual  number  of  adults 
who  are  likely  to  vomit  may  be  approximated  by 
MSI  =  ■  MSDV^ 

where  is  a  constant  which  may  vary  according 
to  the  exposed  population.  For  a  mixed  population 
of  unadapted  male  and  female  adults,  =  1/3  is 
suggested.  The  standards  identify  the  large 
variability  in  the  susceptibility  among  different 
individuals,  e.g.,  females  are  more  prone  to  motion 
sickness  than  males  and  that  the  prevalence  of 
symptoms  declines  with  age.  Therefore,  it  is  noted 
that  Km  should  be  adjusted  accordingly  (Stevens  & 
Parsons,  2002). 

The  base  algorithms  are  also  being  developed 
with  placeholders  for  factors  with  unknown  effector 
values.  The  algorithms  will  obtain  increased  fidelity 


through  iterative  enhancement  as  additional  data  is 
acquired  and  analyses  are  performed. 

Validate  Model  Output 

The  data  validation  plan  includes  identification 
of  HIT  data  type  requirements,  anticipated  data 
sources,  data  collection  methods,  measures  of 
performance,  and  statistical  approaches  to  analysis. 
This  plan  will  serve  as  a  roadmap  for  the 
acquisition,  testing,  and  evaluation  of  algorithms  for 
predictiveness  and  validity.  Models  based  on  HIL 
experimental  and  operational  data  along  with 
algorithm  influenced  modeled  data  will  be  assessed 
for  validity.  These  criteria  include  range  checking, 
causal  dependencies,  temporal  validity,  stability 
check,  cross-validation,  and  SMEs  corroboration. 

Model  predictions  will  be  validated  against  the 
HIL  data  collected  during  performance  of  tasks  in 
an  operational  environment  and/or  laboratories. 
Boring,  Hendrickson,  Forester,  Tran,  and  Lois 
(2010)  recommend  using  three  levels  of  validation 
to  accurately  assess  human  performance  models:  (a) 
successful  task  completion,  (b)  subtask 
correspondence,  and  (c)  quantitative  performance. 
Within  these  levels,  efforts  should  span  from 
coarser,  qualitative  to  finer,  quantitative  levels  of 
granularity.  For  qualitative  validation  (task 
characteristics  such  as  triggering  events,  release 
events,  subtask  composition,  etc.),  analysts  will 
utilize  a  combination  of  SME  input,  consistency 
with  existing  literature,  and  evaluation  of  similar 
models.  Validation  parameters  for  quantitative 
comparison  will  include  time  to  complete  tasks  and 
performance  accuracy  (e.g.,  error  rate,  errors  of 
omission). 

An  inspection  approach  will  be  used  to  check 
that  HIL  and  model  output  data  match  within  the 
benchmark  of  75  ±  10  %.  This  validation  goal  was 
an  estimate  determined  by  a  NAWCTSD  team  of 
human  factors  engineers,  training  specialists,  and 
senior  research  psychologists  with  significant 
experience  in  human  performance  studies  and 
research.  It  was  based  on  past  research  and  sound 
human  performance  research  best  practices  as 
determined  by  the  human  performance  SMEs. 

Based  on  pass/fail  assessment  against  the 
benchmark,  algorithms  will  either  be  adopted  within 
the  PSF  modeling  architecture  or  iteratively 
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recalibrated  until  they  achieve  the  benchmark 
(NAWCTSD,  2011). 

CONCLUSION 

The  purpose  of  this  project  is  to  develop 
processes  for  improving  the  predictiveness  of  PSF 
algorithms  for  environmental  stressors,  either  alone 
or  in  combination  with  other  stressors.  The  goal  of 
this  work  is  to  enhance  the  capability  to  perform 
quantitative  assessments  of  human  performance 
within  affected  operational  environments  (e.g., 
varying  sea  states,  protective  posture,  or  extreme 
heat/cold).  It  is  expected  that  these  predictions  will 
provide  defense  acquisition  leaders,  decision 
makers,  and  design  teams  with  a  more  complete 
picture  of  total  system  performance  and  an 
improved  capacity  to  make  predictions  that  support 
increased  capability,  effectiveness,  safety,  and 
sustainability. 
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